Laboratory tests of manual systems

The essence of manual retrieval systems is that all operations of storage and retrieval are carried out by humans directly, with no more aid than the timehonoured record sheets, index cards, printed pages, and so on. Tests of inhouse systems have covered several kinds of library catalogue, pre-coordinate index, and post-coordinate index. Tests of published systems have compared many styles of printed subject index. In many cases the computer is now used in the construction of both in-house and published systems, but the characteristics of the final product and its use have been affected very little, with the indexing and searching processes still manual in character. Even in systems where a computer is used in matching a formulated search against indexed documents, just as much human skill will be involved, and interactive searching requires human judgement of the highest quality. Though this chapter will concentrate on fully manual systems, some work in these semiautomated areas will be referred to. Laboratory evaluation testing of manual systems began with the first Cranfield project'. With the special library catalogue or index in mind, the traditional index languages (Universal Decimal Classification and Alphabetical Subject Headings) were being challenged by Faceted Classification and the Uniterm system of post-coordinate indexing. So a four-way comparison was mounted. Using a realistically large document collection four test indexes were constructed along practical lines, then laboratory controls were introduced to generate the search requests, identify relevant documents, conduct the test searches, and score the search results. It can be seen now that this approach represents a mid-point in laboratory test techniques between a deeply artificial test of highly controlled subsystems and the testing of real-world systems under conditions of controlled laboratory searching. The deep laboratory approach was soon to be exemplified in the second Cranfield project', where the index language variants tested covered many linguistic forms, and where the searching was so rigorously controlled as to be machine-like or 'unintelligent'. The other extreme of test had begun already with a comparison of two systems by D. R. Swanson, in which one system in particular was analogous to real subject indexes and was subject to